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Foreword 



At Columbia Basin College, the staff of the Office of Institutional Research and Marketing work 
cooperatively with departments to help facilitate different types of assessment projects. We work with 
faculty members on a one-to-one basis and in workshop settings. The CBC Assessment Committee is 
also very willing to help their colleagues in the way of advice and assistance as they begin to study 
different assessment methods and how these can fit into their department’s plan. 

This resource guide came about because of the newness of outcomes assessment to CBC and the need for 
some guidelines and written information about how to develop an assessment plan, the different way to 
assess that plan and how to use the results. The CBC Assessment Committee is also a resource to faculty 
as they establish their outcomes, means of measurement and determining what criteria will spell success. 
Data gathering is an ongoing process for some, periodically or annually for others. A timeline for 
gathering data and presenting reports is included as an appendix in this guide. 

The initial chapter provides guidelines for designing your assessment plan. The second chapter discusses 
the development of departmental goals and objectives/outcomes for learning — ^the basis for assessment 
activities. The remaining chapters concentrate on specific assessment activities - developing tests, 
designing questionnaires, and using institutional data. The focus of each chapter is on the way you can 
use a particular technique for assessment purposes. What can a survey tell you about your program? 

How is a test for assessment different than a course exam? 

Most of the chapters include a question and answer section. Some of the chapters also include exercises 
and checklists. The checklists are not meant to be exhaustive —they are meant to be suggestive. In 
addition, they should allow you to move back and forth through the chapters so that you can focus on 
what’s most relevant to you. 

The staff of the Institutional Research and Marketing Office compiled the workbook. Information for the 
guide came from a variety of resources but most helpful were publications sent to us by Ball State 
University and from a conference workshop administered by the University of Arizona and Arizona State 
University. I sincerely thank them for allowing me to use their materials. 

We hope the workbook is useful to faculty, staff and administrators responsible for assessment. We 
intend for the guide to evolve over time. We welcome your comments and suggestions to increase its 
usefulness. 

The Assessment Committee and the Office of Institutional Research and Marketing are always available 
to work with faculty and staff members on assessment projects and to conduct workshops on assessment 
topics. 



Judi Knutzen, Director 

Office of Institutional Research and Marketing 

Mike Brand, Co-Chair Assessment Committee, 
Math Department 

Pat Campbell, Co-Chair Assessment Committee, 
Student Services 
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Designing a Department Assessment Plan 

This chapter provides guidelines for designing an assessment plan. It suggests 
ways to identify the department's assessment needs and then explores the different 
types of assessment tools and activities that are available to departments. 

In order for assessment to be effective and usefiil, it is advisable to develop 
an assessment plan based on the needs of the department. Having a plan can 
ensure that the selected assessment activities meet the various requirements of the 
department’s constituents, measure what needs to be measured, and provides 
useful data. As assessment plan should address the three basic questions: 

1 . What are we doing? 

2. Why are we doing it? 

3. How will we use it? 



Checklist for Designing an Assessment Plan 

An initial task when designing an assessment plan is to identify the 
department’s assessment needs. The checklist below can be used to answer some 
basic questions about desiring and implementing an assessment plan. 

What are we assessing? 

General education abilities 
Discipline specific knowledge 
Vocational skills 
Success of the program 
Specific courses 

Is there a need to look at assessment requirements of any of the following? 

Curriculum committee requirements 
Professional accreditation 
Professional licensing/certification 
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What do we want to know? 

Knowledge of subject matter 
Skills competency 
Critical thinking skills 
Writing skills 
Attitudes/Satisfaction 
Career success of graduates 

From whom will we collect the data? 

New to college students 
Current students 
Alumni 
Faculty 

Employers of graduates 

Who will see the resulting information? 

Accrediting bodies 

Curriculum committee 

Advisory committees 

Deans and other administrators 

The department 

Students 

Alumni 

Colleagues at other colleges 

How will the data be used? 

Internal discussion 
Curricular revisions 
Reports to constituents 

How often will data be collected? 

Special, one-time projects 
Each quarter 
Annually 
Cyclically 
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Available Assessment Tools 

M^y assessment tools may already be in use within the department. 
Faculty members often engage in evaluative or research activities that are, or can 
be translated into, assessment activities. A review of these processes would 
determine whether they are effective and usable for assessment purposes, and 
whether the department is making the most of existing points of contact with their 
students. New assessment methods may be chosen to fit the needs of the 
department. Several of the assessment tools listed here are discussed in detail in 
the following chapters. 

What existing information is available to the department? 

Trends in student performance 

Tracking exam or course grades over time 

Descriptive indicators for enrolled students or alumni 
Student data such as ASSET, GPA, satisfaction 

Trends in student enrollment 

Retention, enrollment, or graduation rates for students in the 
department 

Course exams or assignments 

Any existing exams, assignments, or projects common to a group of 
students 

Materials describing current curricular program (syllabi, exams, texts, etc.) 

I 

Secondary reading of papers or final exams for critical thinking and/or 
writing skills 

What new information can be generated by the department? 

Surveys 

Entrance surveys 

Why did students select the program? 

Course surveys 

What are student ’s expectations of the course? 

9 - 
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Faculty interviews or surveys 
Alumni surveys 

Employer (of graduates) surveys 

Portfolios - collected examples of student work 

Performance measures - activities such as writing a paper, making an 
oral presentation, completing a computer project, summary 
papers, poster presentations, diagrams/graphics, 
demonstrations, story telling, producing a product, dramatic or 
musical performances, journals/diaries 

Unobtrusive measures - observation and recording of student use of 
facilities and services 

Standardized tests 

Faculty constructed exams 

Minimum competency exams (pre- & post-tests) 
Comprehensive end-of program exams 

Peer feedback/evaluations 

Self-evaluations 

Community member evaluations 

Case studies 

Interviews 
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Qu«stions and Answers on Designing an Assessment Plan 

Q); What ar^. §Qme characteristics of an effective assessment plan? 

A) To effective, an assessment p needs, tp contain the following inapp^nt 
pharapteristics: 

• Assessment should flow from the department’s purpose statenaent and needs 
to be bpth pngping and built into the department’s program. 

• Assessnaent shpuld use nauhiple measures, both qualitative as well as 
quantitative, rather than relying on one instrunaent or aPtivity, 

• A successful assessment program requires fapulty pwnership and 
responsibility since faculty menabers, more than anypne else, are directly 
invplved in the prpcess of student learning. 

• Finally, the results of assessnaent aptivities shpuld lead tP inaprpvenaent and 
be seen as a naeans rather than an end. 

Q) How do we decide where to begin our assessnient? 

A) A gppd place tp start is by examining the depeutnaent’s purppse statement. 
What, accprding to the purppse statement are the intended outepmes of the 
program? Which of these putconaes are most important to the prpgrana? Qnpe 
these intended outconaes have been identified and pripritized, the next stPP is to 
selept the best way to deternaine whether these piitcomes are being achieved. 

Q) Why is it so important to use multiple measures? 

A) The very nature of assessment makes it important that ypu avpid relying pn 
only one measure, such as a test or survey, tp prpvide infonnation at>put ypur 
cpurse pr prpgram. Relying pn one measure is risky since it could result in 
niisleadingperpeptipnspfwhat is going on the department. Students niay 
perform poprly on one test or project, but better on another. Student respppses 
on a survey, while certainly informative, take on a much richer meaning when 
seen with test results from those students. Both pieces are enhanped by 
cprrelation with student database information, Multiple measures allow you to 
explore what students know , what they pan do with what they know, and what 
they think about the whole procpss. Dpn’t forget, too, that important 



information can be gleaned from sources other than students. Data from 
surveys of faculty, alumni, and employers of program graduates can be 
combined with student input to provide a well-rounded picture of your 
program/department. 

Q) How do we involve all faculty in the department? 

A) Some faculty members may be resistant to assessment. A good way to 

overcome that resistance is to involve faculty in the process of assessment. In 
a planning session or focus group format, begin by asking faculty members to 
answer questions such as; “What outcomes do we value?” or “How would we 
identify a successful student in our course(s)?” The resulting discussion can 
spark ideas for an assessment plan. If designing a survey or exam, ask faculty 
members to submit items for the instrument. The most effective assessment 
plan is one that involves and has the support of the department faculty. 

Q) Should we collect assessment data from all our students or just a sample? 

A) This depends on the size of the student population involved. A gen ed course 
that has six sections of from 40-100 students would require only a sample of 
those students to be tested or surveyed. On the other hand, a class that 
typically enrolls only 20 students per quarter might be more effectively 
assessed if the assessment activity involved all the students enrolled. 

Q) Do we need to collect data in all sections of a course or just a sample of 
sections? 

A) Again this depends on the department. There may be several sections of a 
course being taught by the same instructor or using the same text or exams. In 
this case, the department may decide to use only a sample of sections. If, on 
the other hand, several different faculty members using a variety of texts and 
exams are teaching the different sections, it may be wise to use a sample of 
students from each section. The answer to this question also depends on the 
purpose of the assessment project. Assessment of a Math 91 course may be 
focused on exploring whether each section of the course is meeting the goals 
and objectives of the course. This type of assessment will be more effective if 
all sections are involved. A survey of student satisfaction with their learning in 
core courses may require that only a sample of the sections of the course be 
assessed. 






Q) Who will see our department assessment results? 

A) Most assessment data collected by a department is for department use only. 
Assessment provides an opportunity for the department to take a critical look at 
their courses to identify strengths and weaknesses. There are, however, some 
instances where a department is gathering assessment data for a particular 
audience, for example, college accreditation or professional accrediting bodies. 
The Deans of the various divisions should be given copies of the outcomes, 
measurements, and criteria for success and use of the data (assessment records, 
forms B and C) on an annual basis. Some of your assessment data may be 
useful to you in substantiating a budget increase for your department. 

Elements of Outcomes Assessment 
STEPS IN AN ASSESSMENT PLAN 

1 . Program purpose statement tied to college mission statement. 

2. Program goal statement for student learning and development consistent with 
mission statement. 

3. Specific intended learning and developmental outcomes consistent with goal 
statement. 

4. Multiple quantitative and qualitative assessment measures / procedures and 
criteria for assessing each learning or developmental outcome. 

In addition to the above, a successful assessment program requires faculty 
ownership and responsibility. Furthermore, the results of assessment activities 
should lead to improvement and be seen as a means or continuous process rather 
than an end. Examples of those results may be the improvement of teaching and 
learning and/or the evaluation of curricula. As the process is cyclical, the effects 
of changes are then, themselves, evaluated. 

OUTCOMES 

Categories: content learning, cognitive skill development, attitudes and values, 
motivations (e.g., persistence), interests, appreciation, personal, educational, 
social, occupational, economic 
3Rs 

written and oral communication skills 
quantitative and literacy / verbal skills 

13 
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• life-long learning 

• thinking skills - reasoning, critical thinking, problem-solving, creativity 

• knowledge, comprehension, application, analysis, synthesis, evaluation 

• scientific imderstanding / inquiry 

• cultural enrichment 

• social awareness / interaction 

• global consciousness 

MEASURES 

What could be measured to give evidence of the expected outcome? How can it be 
measured? Under what circumstances should we see evidence of the outcome? 
When? What will be seen, done, or possessed when the outcome is reached? 

• Levels of: institutional, general education, departmental 

• Standardized tests 

• Competency exams 

• Locally developed achievement tests 

• Attitude inventories 

• Student and alumni surveys 

• Employer surveys 

• Performance, observation 

• Institutional data 

• License / certification 

• Portfolio analysis: capstone courses; interviews; focus groups 

• Transcript analysis 

• Systematic review of syllabi, textbooks, exams and other curriculum materials 
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Chapter II 

Shaping Department Outcomes for Assessment 

• Goals and Objectives 

• Mastery Versus Developmental Objectives 

• Getting Started 

• Components of Outcomes 

• Drafting Outcomes 

• Checklist for Evaluating Written Outcomes 

• Types of Learning Outcomes to Consider 

• Bloom’s Classification of Cognitive Skills 



Goals and Objectives 



Goals and objectives are similar in that they serve to direct teaching and learning. 
They describe the intended purposes and expected results (outcomes) of teaching 
activities and establish the foundation for assessment. 

Goals are statements about general aims or purposes of education that are 
broad, long-range, intended outcomes. Goals are used primarily in policy 
making and general program planning. 

Objectives are brief, clear statements that describe the desired learning 
outcomes of instruction. Attention is focused on the specific types of 
performances that students are expected to demonstrate at the end of 
instruction. 

Goals express intended outcomes in general terms and objectives express them in 
specific terms. Goals are written in broad, global and sometimes vague language 
while objectives are statements that describe the intended results of instruction in 
terms of specific student behaviors. 

Difference between objectives and outcomes: Objectives are intended results or 
consequences of instruction, curricula, programs or activities. Outcomes are 
achieved results or consequences of what was learned - evidence that some 
learning took place. Objectives specify what is expected and describe what should 
be assessed; outcomes are behaviors and products generated by students after 
instruction and are the objects of assessment. 

Mastery versus Developmental Objectives 

Mastery objectives are typically concerned with the minimum performance 
essentials; those learning tasks that must be mastered by all students for 
success at the next level of instruction. These objectives tend to be limited 
enough in scope that all, or nearly all, intended outcomes can be specified. 

Examples: 

Solve quadratic equations 

Identify symbols used on weather maps 

Identify parts of the microscope 
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Development objectives are concerned with more complex learning 
outcomes - those learning tasks toward which students can be 
expected to show varying degrees of progress. Developmental 
objectives are often written in a two step process in which a 
general objective is stated along with a s^ple of specific learning 
outcomes. 

Example: 



Understand basic theoretical orientations 
State the theory in his/her own words 
Give an example of the theory 

Distinguish between correct and incorrect applications of 
the theory 

Identify predictions that are consistent with the theory 

Example: 



Understand basic scientific principles 

State the principle in his/her own words 

Give an example of the principle 

Distinguish between correct & incorrect application of 

the principle 

Identify predictions that are in harmony with the 
principle 
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Getting Started 

Before writing or revising departmental outcomes, you might try a few of the 
following: 

Have some open discussion sessions on one of the following topics or something 
similar. 

• Describe the ideal student in your program at various phases throughout your 
program. Be concrete and focus on those stren^hs, skills, and values that you 
feel are the result of, or at least supported and nurtured by, the program 
experience. Then ask: 

What does this student know? 

What can this student do? 

What does this student care about? 

• List and briefly describe the program experiences that contribute most to 
the development of the ideal student. 

• List the achievements you implicitly expect of program graduates. 

• Describe your alumni in terms of such achievements as career 
accomplishments, lifestyles, citizenship activities, and aesthetic and 
intellectual involvement. 

Collect and review instructional materials. Try sorting materials into three broad 
caitegories: recbgnition/recall, comprehension/simple application, critical 
thinking/problem-solving. Use any of the following: 

• Syllabi and course outlines. 

• Course assignments and tests. 

• Textbooks (especially the tables of contents, introductions, and 
summaries). 

Collect and review documents that describe your department/program: 

• Brochures and catalogue descriptions 

• Accreditation reports 

• Curriculum committee reports 

• Mission or purpose statement 

18 
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Review and react to outcomes from another department that is similar but external 
(example, another community college in Washington). Try grouping the 
statements into broad categories of student outcomes (i.e., knowledge, attitudinal, 
behavioral). 

Use the 25 percent rule to refine or reduce a set of goal statements. Imagine that 
you want to reduce program or course material by 25 percent. What goals would 
you keep and which you discard? 

Administer a goals inventory or conduct an interview study. Involve a variety of 
groups (or “stakeholders”) when possible. 

Use a Delphi technique or a modification. This involves administering a series of 
related questionnaires in which information from the initial form is provided so 
that respondents can use it to revise their responses on subsequent forms. The 
objective is to develop consensus before writing your outcomes. 

Components of Outcomes 

The essential and optional components of outcomes are described below. The 
essential elements are listed first. After a component is defined, the phrase that 
illustrates it will be extracted from the following outcome. 



After analyzing and interpreting information from public opinion polls, 95% of the 
second quarter marketing students will be able to communicate the results in at 
least three different forms - written, oral, and graphic. 



Essential components 

Behavior: specify actions or behaviors that follow instruction and could serve as 
evidence that the object has been achieved; use active verbs that describe 
observable behavior. 

Example: communicate results 

Object: Identify the focus of learning-content, concept(s), skill, or attitude. 
Example: public opinion polls. 

Criteria for success: 

Example: 95% 
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Target groups: specify subgroups when objective applies differently 
Example: second quarter marketing student 

Conditions: give information about situations in which the student will be required 
to demonstrate die behavior — ^how, when, or where 

Example: after analyzing and interpreting information 



Performance criteria: state any minimum level of performance 
Example: in written, oral, and graphic forms 



20 
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Worksheet 1: Drafting Outcomes 

Department and/or Course: 

Part 1. Place keywords in each of the component cells. 



Components 







Target 


Performance 


Criteria for 


Behaviors 


Objects 


Group 


Condition/s Criteria 


Success 



1. 












2. 












3. 












4. 












5. 













Part 2. Write statements that combine the components into meaningful objectives. 



1 . 

2 . 

3. 

4. 

5. 
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Checklist for Evaluating Written Outcomes 
The outcomes statement. . . 

• Uses action verbs that specify definite, observable behaviors. 

• Uses simple language. 

• Describes student rather than instructor behaviors. 

• Describes a learning outcome rather than a learning process. 

• Focuses on end-of-instruction behavior rather than subject matter coverage. 

• Indicates a single outcome per statement. 

• Can be assessed by one or more indicators (methods). 

• Is clearly linked to a goal. 

• Is realistic and attainable. 

• Is not simple when complexity is needed. 

• Is clear to people outside the discipline. 

• Is validated by departmental colleagues. 






19 



Types of Learning Outcomes to Consider 

Gronlimd (1981) provided the following list of types of outcomes. The list 
delineates many of the major areas in which instructional objectives might be 
produced. The specific categories were intended to be suggestive, not exclusive. 

Knowledge 

• Terminology 

• Specific facts 

• Concepts and principles 

• Methods and procedures 

Understanding 

• Concepts and principles 

• Methods and procedures 

• Written material, graphs, maps, and numerical data 

• Problem situations 

Application 

• Factual information 

• Concepts and principles 

• Methods and procedures 

• Problem-solving skills 

Thinking skills 

• Critical thinking 

• Scientific thinking 

General skills 

• Laboratory skills 

• Performance skills 

• Communication skills 

• Computation skills 
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• Social skills 



Attitudes 

• Social attitudes 

• Scientific attitudes 

Interests 

• Personal: interests 

• Educatioinal interests 

• Vocational interests 



Appreciation 



• Literature, art, and music 

• Social and scientific achievements 

Adjustments 

• Social adjustments 

• Emotional adjustments 



Bloom’s Classification of Cognitive Skills 

Bloom’s classification of cognitive skills is widely used in instruction planning. 
The six levels are arranged by level of complexity. Use of this or other 
classification systems is recommended to safeguard against a tendency to focus on 
content coverage and to ignore what the students should learn to do with content. 

Category Definition Related Behaviors 



Knowledge 


Recalling or remembering 
something without necessarily 
understanding, using, or 
changing it 


Define, describe, identify, 
label, list, match, memorize, 
point to, recall, select, state 


Comprehension 


Understanding something that 
has been communicated 
without necessarily relating it 
to anything else 


Alter, account for, annotate, 
calculate, change, convert, 
group, explain, generalize, 
give examples, infer, interpret, 
paraphrase, predict, review, 
summarize, translate 


Application 


Using a general concept to 
solve problems in a particular 
situation; using learned 
material in new and concrete 
situations 


Apply, adopt, collect, 
construct, demonstrate, 
discover, illustrate, interview, 
make use of, manipulate, 
relate, show, solve, use 


Analysis 


Breaking something down into 
its parts; may focus on 
identification of parts of 
analysis of relationships 
between parts, or recognition 
of organizational principles 


Analyze, compare, contrast, 
diagram, differentiate, dissect, 
distinguish, identify, illustrate, 
infer, outline, point out, select, 
separate, sort, subdivide 


Synthesis 


Creating something new by 
putting parts of different ideas 
together to make a whole 


Blend, build, change, 
combine, compile, compose, 
conceive, create, design, 
formulate, generate, 
hypothesize, plan, predict, 
produce, reorder, revise, tell, 
write 


Evaluation 


Judging the value of material 
or methpds as they might be 
applied in a particular 
situation; judging with the use 
of definite criteria 


Accept, appraise, assess, 
arbitrate, award, choose, 
conclude, criticize, defend, 
evaluate, grade, judge, 
prioritize, recommend, 
referee, reject, select, support 
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Chapter III 

Using Surveys for Assessment 

Surveys About the Department 

Surveys for General Education Assessment 

Item Construction Cautions and Concerns 

Examples of Flawed Survey Questions 

Questions and Answers About Using Surveys for 
Assessment 



Using Surveys for Assessment 

The purpose of this chapter is to provide suggestions for designing and using 
surveys for assessments. Included are guidelines for constructing effective surveys 
as well as suggestions for how different types of surveys can meet the various 
assessment needs of a program or department. 

A comprehensive assessment plan provides an opportunity for students, faculty, 
and alumni to express their perceptions of a course or program. Accrediting bodies 
frequently ask departments to provide input from students and alunmi. Surveys are 
an excellent way of collecting this type of information. A survey can be locally 
constructed or purchased. It can be designed for specific groups such as students 
in a particular course, or alumni of the program. Surveys can Ije one page or 
several pages. They can be given in class or mailed. A variety of information can 
be obtained from a survey, as illustrated in the checklist below. You can choose 
which types of information will provide the most complete picture of the group 
you plan to survey. 

Checklist of types of survey information 

• Attitudes and opinions 

• Values 

• Experiences 

• Expectations 

• Needs 

• Demographics (gender, race, age, residence, marital status) 

• Descriptive data (gpa, ASSET scores, etc.) 

Surveys About the Department 

As part of its assessment plan the department/program may want to target a survey 
to a specific audience such as entering students, current students, exiting 
sophomores, alumni, faculty, or employers. Each group can provide specific types 
of information as suggested below. 

Suggestions for Using Surveys 

• Entering student surveys may be administered to students entering the program 
or entering a course. Surveys for entering students may ask questions such as 



why students selected the program or course and what their expectations are for 
the program or course. 

• Current student surveys may be administered to all students in the program or to 
a sample of students in selected courses. These surveys can help determine the 
perceptions and attitudes of students currently in the program. 

• An exiting student survey allows the department to gather comments from 
graduating sophomores concerning their perceptions of the department, their 
immediate and future plans, and if they have been able to obtain employment or 
been accepted into a four-year college or university major. 

• Alumni surveys allow departments/programs to determine the types of jobs 
their graduates are able to obtain and the skills that are necessary for the job 
market. These surveys provide information about entry level salaries, as well as 
perceptions of the department/program after being away for a time. Alumni can 
also reflect on the knowledge and skills gained in the program. 

• Faculty surveys allow the department to determine if there is consensus on the 
goals and objectives of the department, if the faculty members feel the goals 
and objectives are being met, the strengths and weaknesses of the program, and 
faculty members’ expectations of students entering the program. 

• Employer surveys enable the department/program to determine if their 
graduates have the necessary job skills and if there are additional skills, which 
are especially sought after or give their graduates an extra “edge” in gaining the 
job. 

Possible Survey Questions 

There are several types of data that can be obtained on a survey instrument. The 
department may be interested in finding information about students’ perceptions of 
the department, levels of involvement, perceived competence at various skills, 
students’ satisfaction, employment or education status, and faculty opinions about 
the department. 

The following are examples of questions that may be included on a survey used for 
the purpose of assessing the department. 




25 

28 



Entering Students 

Why did you select this program? 

What are your immediate academic/vocational goals? 

What do you expect to gain from this program? 

Current students 

Do you participate in study sessions? 

Do you participate in departmental activities? 

How satisfied are you with the program/department curriculum? 

Exiting sophomore 

How satisfied are you with your program? 

How well prepared are you for the future? 

Are you planning to transfer to a four-year institution? 

Are you planning to go to work using your program skills? 

Alumni Surveys 

Are you employed full-time? 

Are you employed in the area of your program? 

Are you currently enrolled at a four-year institution? 

Faculty Surveys 

What do you see as the strengths of the department? 

What do you see as the weaknesses of the department? 

Employer Surveys 

Does the employee have the necessary skills for the job? 

In what area is the employee best prepared? 

What are areas the program should emphasize to ensure viable candidates 
for a position in your field? 

Degree Requirement Course Surveys (Gen Ed) 

The main purpose of using surveys, for assessment of courses required for a degree 
or courses considered general education courses, is to determine if students and 
faculty feel that the goals and objectives of the course or degree are being met. A 
survey administered in a required course has questions that are about the course 
rather than the instructor. 
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Questions concerning the students’ background, study techniques, academic 
performance, content evaluation, and faculty opinions and perceptions can provide 
information about these courses. 

The following are examples of questions that can be included on a survey used for 
required courses or courses that are considered general education courses: 

Student Background 

How many high school courses did you take in this subject? 

How would you describe your academic preparation for this course? 

What was your high school GPA? 

Is this the first time you have taken this course? 

Study Techniques 

How many hours a week did you spend studying for this course? 

How often did you study for this course with someone else? 

Student Performance 

What grade do you presently have in this course? 

How many times did you miss this class this quarter? 

How would you describe your efforts in this course compared to other 
courses? 

Content Evaluation 

Did this course significantly stretch and broaden your world view? 

Was this course effective in improving your understanding of your 
field of interest? 

Was this course effective in improving your understanding of the subject? 
Faculty Opinions and Perceptions 

How would you define the goal and objectives (learning outcomes) of the 
course(s)? 

Do you feel program goals and objectives (outcomes) are being met? 

Item Construction Cautions and Concerns 

When constructing a survey, make sure survey items and response categories: 

• do not lead the respondent into giving an answer that he/she would not 
ordinarily give. The survey items should be stated in a neutral manner. 
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may not be interpreted in more than one way. 
are not vague, 
are not too precise, 
are not too personal. 

are not likely to be endorsed by almost everyone or almost no one. 
have words that are simple, clear, direct, and uniformly understood, 
are short, rarely exceeding 20 words. 

contain only one question - avoid double-barreled questions, 
avoid words such as “only,” “just,” and “merely.” 

are in the form of simple sentences rather than compound or complex sentences, 
avoid the use of double negatives, 
are not hypothetical. 

avoid abbreviations or unconventional phrases, 
avoid unequal comparisons among response categories. 



Examples of Flawed Survey Questions 



The following worksheet has survey items with item construction problems as 
described on the previous page. Which of the following types of errors occur in 



the exercise? 


■ vague question of response 


■ question is too precise 


categories 


■ question that is too personal 


■ question or answer categories that 
are biased 


■ double barreled question 


■ double negative question 


■ overlapping response categories 


■ complex wording, abbreviations 
or unconventional phrases 



1 . How often did you attend sporting events during the past year? 

1. Never 

2. Rarely 

3. Occasionally 

4. Regularly 

Problem: Vague answer categories 
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2. How many books did you read last year? 

Number 

Problem; Too much precision 

3. Most people have a good understanding of the Big Bang Theory. Do you have 
a good understanding of the Big Bang Theory? 

1. Yes 

2. No 

Problem: Bias 

4. Who do you feel is most responsible when a student fails in college? 

1. The student 

2. The parents 

3. Unsympathetic faculty members 
Problem: Bias, unequal comparison 

5. How much money are you making? 

Dollars 

Problem: Too personal 

6. Should the course HS 1 04 not be offered after the year 1999? 

1. Yes 

2. No 

Problem: Double Negative 

7. How many plays or movies have you attended this semester? 

1. 0-3 

2. 3-6 

3. 6-10 or more 

Problem: Double-barreled and overlapping categories 
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Questions and Answers about Using Surveys for Assessment 

Q) How do we decide which group to target for a survey? 

A) The answer to this question depends on who your audiences is and what they 
need to know. Do you want to know how students learned about your program 
and what they expect from it? In this case, a survey of students just entering 
the program is appropriate. Are you interested in exploring students’ 
perceptions of the program and how well it met their expectations? Surveying 
students about to graduate would give you this information. Accrediting 
agencies sometimes require information about department/program alumni. 

Q) How do we generate appropriate questions for a survey? 

A) One suggestion would be to ask faculty members to submit survey items — 
what would they like to learn from the students? Some departments designate 
a committee or even one-faculty member to design a survey. A good place to 
start is by looking at surveys that have been done in departments similar to 
yours. Commercially prepared surveys are available from places such as the 
Educational Testing Service (ETS) or the American College Testing Program 
(ACT). 

I . 

Q) How can we ensure a good response rate on our surveys? 

A) One of the most crucial factors affecting response rate is the length of the 
survey. A survey should only take about 15-20 minutes to complete. 
Appearance is, of course, important and the survey should be easy to fill out, 
with clear and concise instructions. A cover letter that explains the purpose of 
the survey and assures the confidentiality of the respondents’ answers can 
make a difference in whether a survey is completed and returned. Having the 
letter sent by someone they know and respect is important as is enclosing a 
self-addressed envelope. If you have problems getting a good return rate, 
contact the Institutional Research and Marketing Office for information on a 
4-step process that raises response rates. 

Q) Should our surveys be anonymous? 

A) Whether or not you ask students to put their names on surveys depends in part 
on whether the surveys are to be mailed or given in class. Asking students in 
class to put their names on surveys could make students uncomfortable and 
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reluctant to respond candidly. Also the human subjects policy requires that 
student’s know they do not have to participate if they so choose. A mail 
survey is less personal and, therefore, easier to ask for an identifier, such as a 
social security number. However, again the students must be told that giving 
this information is optional. There are several advantages to identifying 
respondents. First, identifying the respondents allows you to do a follow-up 
mailing and thus increase your response rate. Secondly, identified surveys 
allow you to match the survey data to other information that helps you describe 
the academic and demographic characteristics of the respondents. When the 
respondent is identified, be sure to stress that survey responses will be kept 
confidential. 

Q) Should we do a follow-up mailing? 

A) Follow-up mailings can be helpful in boosting the return rate - often by as 
much as ten percent or more. However, they do require that the respondents be 
identified either through a numbered label on the survey or return envelope so 
someone who has responded will not receive a survey twice. A follow-up 

mailing will also increase the cost of doing the survey. 

A 

Q) What are the costs involved in doing a survey? 

A) Copying and mailing are the biggest expenses for a mail-out survey. A 

department may choose to do their own copying or use the campus print room. 
The cost of copying is affected by several factors such as the number of 
surveys and the style and lenglii of the survey. Mailing costs vary depending 
on whether you choose to send the surveys bulk rate or first class, whether you 
include a postage-paid return envelope, as well as the size of the envelope. 
Departments sometimes choose to reduce the cost of surveying students by 
administering surveys in class or making them available in accessible locations 
such as the department office. 

Q) Should we survey the entire group or just a sample of the group? 

A) In deciding whether to survey all your students or a sample, you need to 
consider factors such as cost, available support staff, and time constraints, as 
well as your assessment needs. Your choice will depend in part on the size of 
the student population involved. A survey of students in a course that has 
several sections would probably require only a sample of students. Contact the 
Institutional Research and Marketing Office for guidelines on sample size. 
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Q) Should we use thei traditional form of survey or one with scan sheets or a 
scanable survey form? 

A) Using the traditional survey, in which students put their answers directly on the 
survey rather than on a scan card, has the advantage of allowing for open- 
ended responses. Students generally prefer this style of survey since it seems 
more personal, less like an exam. Scanable forms require a #2 pencil and need 
to be filled out very carefully by the respondents. On the other hand, scan 
sheets and cards allow for quicker data analysis, and do not require support 
staff to handle the data entry. Using scan cards and scanable survey forms is 
not recommended for mailed-out surveys since they generally cannot be 
folded. For more information on survey forms built to be scanable and which 
the Scantron reads, contact the Institutional Research and Marketing Office. 
This office has the software and equipment available to create scanable forms. 
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Chapter IV 

Using Tests for Assessment 

• Questions and Answers About Using Tests for 
Assessment 

• Features of Various Test Forms 

• Test Blueprint 

• Components of a Test Blueprint 

® Steps for Completing a Test Blueprint 

• Potential Resources for Locally Constructed Tests 

• Suggestions for Reviewing an Externally-Constructed 
Test 

• Item Writing Suggestions 

• Quality of Test Items 

® Calculating Item Difficulty and Discrimination 
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Using Tests for Assessment 

7%w phapter provides some guidelines for using locally or externally developed 
tests for assessment. Tips for planning and developing a test as well as for 
analyzing the quality of a test are included. 

Questions and Answers About Using Tests for Assessment 



Q) How does la test fit into an assessment plan? 

A) In most cases, a test will be one part of a fully developed assessment plan. 
Most progr^s have cognitive, attitudinal, and performance goals. Tests are 
cprnmonly used in association with cognitive goals - to review student 
achievement with respect to a common body of knowledge associated with a 
discipline or practice. Tools such as interviews and surveys are used more 
commonly in conjunction with other non-cognitive assessment techniques. 

Q) When is a test a good choice for an assessment program? 

A) Use of a test is a good choice when; 

• The student is to demonstrate acquisition of knowledge or ability 
to process and use knowledge (i.e., the outcome of interest is 
cognitive in nature). 

• The student’s knowledge about a wide range of content is to be 
evaluated (for example, survey courses). 

• Multiple observations of the content-related knowledge are needed 
(for example, math and foreign languages). 

• A large group is being assessed. 
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Q) What are the major advantages of testing as an assessment technique? 

A) In comparison to other assessment procedures, tests can sample student 
knowledge with efficiency and reliability - you can find out what a lot of 
students know in a brief period of time. A second advantage is that repeated 
use of a test will provide a means of comparison between different student 
groups or the same group over time. This type of testing practice provides 
faculty with a rich context for evaluation, decision-making, and 
recommendation. 

Q) What are the distinguishing features of a good test? 

A) At a minimum, a good test will have: 

• A well-defined purpose or intent. 

• A foundation based on a set of written goals and objectives. 

• Some evidence that the test purpose was achieved - reliability and 
validity information about test items, the test as a whole, and the 
relationship between test scores and other indices of academic 
performance. 

• A detailed scoring procedure that allows for specific interpretations 
and feedback to those tested and those making the decision. 

Q) What are some strategies for creating a departmental test for assessment? 

A) One common practice is to develop and adopt common pretests and post-tests 
in courses with multiple sections. Items for common tests could be culled ffOm 
existing exams. Another practice is to determine a portion of each unit exam, a 
specific set of items, that are scored for program assessment and also for 
individual evaluation. This practice is sometimes referred to as course- 
embedded testing. (When using course-embedded testing, it is important to 
notify students how the practice affects the assignment of grades.) 

Q) How is using a test for assessment purposes different from using a test in the 
classroom? 

A) Generally, instructors develop their own classroom tests, making all decisions 
about when and how to construct, administer, score, and report results of tests. 
Construction is often done without formality or documentation. When 
assessment, planning, implementing, and using results becomes a group effort. 



a steed set of decisions and responsibilities occurs. Consensus is emphasized. 
Some additional planning time, communication, and record keeping will be 
heeded. The most frequent use of tests of instructors is to assign grades related 
to ittdividual student learning. When used for program assessment, test 
perfdniriahce is generally used along with other information to describe group 
achievement ahd is independent of grading. 

Q) Mow can we prepare students for tests as assessment? 

A) Because tests for assessment are different than tests for grading, it is 

recommended that you provide a brief orientation which covers the following 
topics: 

- • The purpose of the test and how results will be used 

• How the students will benefit 

• Whether guessing is appropriate 

• Whether names and identification numbers are needed and why 

Q) What is a “standardized” test? 

A) A standardized test is one in which the initial construction, as well as 

conditions for administration and scoring, have a uniform procedure so that the 
scored may be interpreted in a Consistent manner from one administration to 
the next. Test development specialists design these, either internally or 
extemally. 

Q) What are the basic steps in developing a test? 

A) Seven basic and sequential steps are recommended: 

• Determine the outcomes to be measured. 

• Develop a test blueprint. 

• Write the test items. 

• Review, critique, and edit the items. 

• Pilot the items. 

• Obtain reliability and validity data. 

• Revise, reuse, and report. 
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Q) What are the differences between a locally-developed and an externally- 
developed test? 

A) Differences between locally- and externally-developed tests are described 
below. 

Characteristics Externally-Developed Locally-Developed 



Tests Tests 



Development time 


None 


Varies; depends on local 
testing practices, test 
development resources 
and expertise 


Relationship between test 
and program objectives 


Varies; test is tailored to 
broad-based needs 


Close; tailored to local 
needs; adjusted as needed 


Comparison groups 


May include national, 
regional; may include 
norms by gender, class 
level, college, institution 
type; infrequently updated 


Created and maintained 
locally; generally no 
external norms; can be 
modified as needed 


Costs 


Usually high; materials 
and scoring costs may be 
reoccurring 


Usually low; can be 
managed with limited 
reoccurring costs 


Results 


May be long delays; little 
choice in type of analysis 


Can be immediate; local 
needs/decisions drive 
analyses 



Q) Are there any important decisions that precede test development? 

A) Yes. Before a test is developed, a few decisions about use of results need to be 
clarified because they will have an influence on test construction. Three 
critical decisions are raised here. 

1. Are group or individual results needed? Good questions to ask are: 

• Can individual scores be created from group scores? 

• Are individual-level data necessary for programmatic decisions? 

• Do the decisions being made affect individuals? 
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2. What types of item formats are needed? Items can be broadly classified as 
either recognition or production types. A detailed coniparison of types of 
tests is provided in Table 1 . Good questions to ask are: 

• Dbes the iteni format allow inferences about the thinking skills of 
interest? 

• Is there a simpler format for the information of interest? 

3. Are cnterion- or norm-referenced results needed? Good questions to ask 
follow a brief description of each type of test. 

Norm-referenced tests: 

• Provide relative comparisons. 

• bescribe performance in terms of the relative position held by 
some known group or individual. 

• Are used to judge performance in comparison to others. 

Criterion-referenced tests: 

• Provide absolute comparisons. 

• Describe the desired student performance directly and in detail. 

• Are used to judge performance by some pre-set standard 
(proficiency or mastery). 

Good questions to ask are: 

• Is mastery Of information Or thinking skills necessary to mbVe to a new 
area of study? 

• bo all students need to show a minimal level of knowledge? By a certain 

time? 

• bid studeiits begin with a wide range of ability? 
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Table 1: Features of Various Test Forms 



Objective Essay Oral Performance 



Conditions for 
Use 


Assess knowledge 
w/maximum 
efficiency & 
reliability 


Assess thinking 
skills and/or 
mastery of a 
structure of 
knowledge 


Assess knowledge 
during instruction 
process 


Assess ability to 
use knowledge to 
create a new 
product 


Stimulus 

Material 


Multiple-choice, 
True/False, Fill-in, 
Matching 


Writing task 


Open-ended 

prompts 


Event or directions 
that provide a 
frame for a 
performance or . 
product 


Student’s 

Response 


(Recognition) 
Select from 
options provided 


(Production) 
Organize, 
construct, & 
supply 


(Production) 
Interpret, 
construct, & 
deliver 


(Production) Plan, 
construction & 
deliver original 
response 


Scoring 


Count correct 
answers 


Judge 

understanding 


Determine 
correctness of 
answer 


Apply attributes, 
checklists or rating 
scales to describe 
performance or 
proficiency 


Major Strengths 


Efficiency — can 
sample broad 
content range 


Gets at complex 
thinking 


Immediately links 
assessment and 
instruction 


Provides rich 
evidence of 
performance skills 


Potential 

Weaknesses 


Poorly written 
terms, over- 
emphasis on 
factual recall, poor 
test-taking skills, 
failure to obtain a 
representative 
sample of content 


Poorly written 
exercises, 
confounded with 
knowledge of 
content, poor 
scoring procedures 


Poor questions, 
students lack of 
willingness to 
respond, too few 
questions 


Poor exercises, too 
few samples of 
performance, 
vague criteria, 
poor rating 
procedures, poor 
test conditions 


Influence of 
Format on 
Learning 


Can require 
complex thinking 
skills but may 
focus on simple 
thinking skills 


Encourages 
thinking and 
development of 
writing skills 


Stimulates 

participation in 

instruction, 

provides 

immediate 

feedback 


Emphasizes use of 
skill & knowledge 
application in 
problem context 


Keys to 

Successful Use of 
Format 


Clear test 
blueprint, item 
writing skill, 
ample construction 
time 


Carefully prepared 
writing prompts & 
model answers, 
ample scoring time 


Clear questions, 

systematic 

sampling 

procedures, ample 
time for response 


Clear performance 
criteria, clear 
rating scales, 
ample rating time 



Adapted from: Stiggins, R. J. (1987), p. 35. 
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Test Blueprint 



Tests that are soundly constructed are built to meet specifications just as a house is 
built according to a plan. The plan for a test is frequently called a test blueprint 
(also ca!lled a table of specifications or a test matrix). Two dimensions are laid but 
as the columns and rows of a two-way table. At the intersections betweep the rows 
and columns, the number of items needed to test the area (or percentage of 
importance) is recorded. A simple example is provided in Figure 1 . 

Figure 1 . Example of a Test Blueprint for an Early American History Test 
Discipline/Topic: Early American History 



Content Process: Levels of Thought Total 



Historical 

Periods 


Recall Facts 


Comprehend 

Concepts 


Apply Facts & 
Concepts 




Exploration 


10(20%) 


5 (10%) 


1 (2%) 


16(32%) 


Colonization 


10 (20%) 


5 (10%) 


1 (2%) 


16(32%) 


Revolution 


12 (24%) 


5 (10%) 


1 (2%) 


18(36%) 


Total 


32(64%) 


15(30%) 


3 (6%) 


50(100%) 



Things to hbtiee in the blueprint shown in Figure 1 . 



• The distribution of items by content areas is relatively equal although there is a 
slightly greater ertiphasis on Revolution - 36% versus 32% in each of the other 
two areas. 

• The process that appears to be most important is Recall of Facts - 64% versus 
3 G% or 6% of the items. 

• Because of the fairly equal distribution of emphasis on three content areas and 
the heavy emphasis on knowledge of factual information, this test might be 
used early in a term as a review or as a pretest with new learners. 
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Components of a Test Blueprint 



Content areas, thinking processes, and importance specifications are given in test 
blueprints. Using the steps below and the blank test blueprint (Exercise 1) a 
practice test blueprint can be drafted. 

Exercise 1 . Creating a Sample Test Blueprint 

Discipline/Topic: 

Content Process: Levels of Thought Total 



1: 2: 3: 



1. 


( %) 


( %) 


( %) 


( %) 




( %) 


( %) 


( %) 


( %) 


2. 




( %) 


( %) 


( %) 


( %) 


3. 














Total 


( %) 


( %) 


( %) 


( %) 
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Steps for Completing a Test Blueprint 

1 . First, identify the discipline/topic area to be addressed by the test in the blank 
provided. 

2. Next, define the test’s content areas. Along the left column, list at least 3 
general content areas from your discipline. 

3. Now, define the intellectual processes to be assessed. Along the top rOw, list at 
least three intellectual processes. 

4. Finally, define the importance of the test content areas and processes to the 
overall test. Enter the number and/or percentage of itenis in the blanks where 
content areas and processes overlap. The values should reflect the importance 
of content/process relationships as conveyed in the curriculum and instruction. 

Suggestions for Developing Test Blueprints 

• Refer back to the goals and objectives written for the assessnlent plan. 

• Assign priorities to goals and objectives so that the most important content 
areas and thinking skills are addressed by your test. 

• Remember that a test provides a sample or samples of behaviors and should be 
conibined with other information about performance. 

Functions of a Test Blueprint - A Summary: 

• Helps achieve balance between instruction and assessment. 

• Reduces tendency to test “memory of facts” only. 

• Helps ensure that a test will sample all important content and process areas. 

• Provides a structure for communicating with students before and after testing. 

• Provides structure for analyzing, summarizing, and reporting results. 
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Potential Resoinrces for Locally Constructed Tests 

Internal: 

• Unit tests or finals from individual sections of a course 

• Departmental tests - retired and current 

• Student generated items - however, the items usually need revision 

External: 

• Curriculum materials that accompany textbooks may provide unit/chapter tests 
and item banks. 

• Professional agencies associated with a discipline may have retired tests or item 
banks related to certification or licensing exams. 

• Colleagues teaching at other institutions may have developed instruments/items 
for similar needs. 

• Test companies sell published instruments and item banks. 

• Journals that focus on teaching of the discipline sometimes contain unpublished 
tests. Social science and education journals that focus on instructional practice 
are good sources. 

Suggestions for Reviewing an Externally-Constructed Test 

• Remember that it is unlikely there will be 100% match between an externally- 
constructed test and local curriculum and instructional practice. 

• Before adoption, all tests need to be examined in terms of: 

• The match between the purpose for which the test was designed 
and the local needs based on stated student learning outcomes 

® The correspondence between the test blueprint and content of 
related items 

• Review the technical merits of the instrument. Look and/or ask for the 
following: 

o Norms/technical manual 

• Test blueprint 

® Information about item development 

• Evidence of reliability and validity 

• Relatively current norms, information about norm groups 
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» Pilot the instrument with a small group before commitment to full-scale use. 

• Call other users - publishers should provide a list of contacts. 

• Always check costs before use: usable/non-reusable materials, scoring (local or 
external), and reports. 

Item Writing Suggestions 

This section contains general item writing tips as well as specific tips for various 

item types. 

Suggestions for Ail Item Types 

• Write items directly after instruction when possible. 

• Write more items than are needed - 25% more is a good rule of thumb. 

• Keep reading and vocabulary levels simple unless you are testing those skills 
(i.e., avoid superfluous wording). 

Item. Stems (definition: the part of a test item that poses the question or sets up the 

problem situation; the stimulus) 

• Make sure the problems posed in the item stems are clear and unambiguous. 

• Use “concrete” situations and pictorial, graphic, or tabular stimuli when 
possible. 

• Use novel material in formulating problems to measure higher level mental 
processes (i.e., analysis and application type items); however, guard against 
overuse of this strategy. 

• Avoid lifting statements verbatim fi'om a text unless your intention is to test 
memorization skill or knowledge for siniple and basic information. 

• Avoid single and double negative items; phrase in a positive format if possible. 

• Be careful of clues and ^swer cues within the item stem. 

item Responses 

• Avoid noun modifiers like “all,” “never,” “no,” and “always.” 

• Order the response options in some parallel way (e.g., alphabetically, 
chronologically), but avoid creating response patterns (e.g., an alternating true- 
false pattern, overuse of the “C” option in a set of multiple-choice items). 
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Suggestions/Checks for Multiple-Choice Items 

• Avoid highly technical response options. 

• Avoid having the correct answer longer than the incorrect options. 

• Use responses that are plausible and homogeneous in some way. 

• Use between three and five options. Try to use the same number of options for 
all items; however, do not create superfluous options just to maintain a parallel 
format. 

Suggestions/Checks for Matching Items 

• Provide more choices than number of statements to be answered unless a choice 
can be used more than once. 

• Have students choose answers from the column with the least amount of 
reading. 

Suggestions/Checks for True-False (Alternative Response) Items 

• Avoid ambiguous and indefinite terms of degree or amount (i.e., “frequently,” 
“in most cases,” etc.). 

• Avoid negative and double negative statements. 

o Keep true and false statements approximately the same length. 

• Have approximately the same number of true and false items on a test. 

Suggestions/Checks for Writing Completion or Short Answer Items 

• Draft items that require a single-word answer or a brief and definitive 
statement. 

• Avoid statements that may be logically answered by several terms. 

• Indicate the unit of expression (i.e. , date, percentage) when answers require 
numerical information. 

Suggestions/Checks for Writing Essay Items 

• Select items carefully because of the limited number that can be given in a 
single time frame. 

• Make items clear and specific so that scoring can be done easily. 

• Establish a framework within which the student will write. 
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1. Omit the ^ea covered by 

2. Indicate the value of items and sugge$t time parameters, 

3 • Decide, in advance, the factors considered in evaluation and note 
them in the instructions. 

Quality of Test Items 

Analyzing the quaH items can be accomplished in phases. Some general 

suggestions foilow: 

1 . Before #ie test is used (pilot tested), ask faculty members to check for the 
following: 

Congruence between test content and test blueprint 

• Do items reflect the speeified topic weights? 

• Do items measure the range/emphasis of cognitive abilities 
speeified? 

Clarity of directions 

XJse of good item construction practices 

Accuracy, specificity, and emphasis of scoring/rating procedures 

2, After a test is given for the first time (pilot tested) refer to the following 
checklists: 

If using an objective test format, check for: 

• Item difficulty 

• Item discrimination 

If using other formats, check for: 

• Consistency of application of scoring criteria by those doing the 
rating 

Have a sample of responses scored by a second evaluator. 
Have a sample of responses scored in a different order and at 
a different time. 
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Calculating Item Difficulty and Discrimination 

Before test scores are interpreted or used for decisions, it is essential to review how 
well the items functioned. Calculating item difficulty and discrimination values is 
recommended practice. 

Difficulty Index (P value): 

An items difficulty index is expressed as the proportion of students who responded 
correctly (successfully) to an item. If scores from all students in a group are 
included, the difficulty index is simply the total percent correct. When there is a 
sufficient number of scores available (i.e., 100 or more) difficulty indexes are 
calculated using scores from the top and bottom 27 percent of the group. 

The value is interpreted in ^ inverse way, that is, a high value is interpreted as less 
difficult. These values range from 0 to 1.00 and are usually expressed as a 
percentage. 

Item difficulty is calculated in the following way: 

P = Successes in the HSG + Successes in the LSG 
NinHSG + LSG 



(HSG=high scoring group 
LSG=low scoring group 
N=number) 



Item difficulty is generally interpreted in the following way: 


P- Value 


Percent Ranee 


Interpretation 


> or = .75 


75-100 


Easy 


< or = .25 


0-25 


Hard 


between .25 & .75 


26-74 


Average 



Example: 

Item: Who is the current host of the Tonight Show? 

A. Johnny Carson 

B. Arsenio Hall 

C. David Letterman 

D. Jay Leno 
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Groups A B C D* Total 



High Scorers 0 1 1 8 10 

Low Scorers 1 1 5_ 3_ 10 

Total 1 2 6 11 20 

P = (8+3)/(10+10) = 1 1/20 = .55 

Interpretation: This item is average in difficulty. Slightly more than 
half of the students got the item correct. 

Discrimination Index (D value): 

An item’s discrimination index is expressed as the difference between high and 
low scorers. The value is interpreted in terms of both direction (positive or 
negative) and strength (non-discriminating to strongly discriminating). These 
values can range from -1 .00 to + 1 .00. 

Item discrimination is calculated in the following way: 

D = Successes in the HSG - Successes in the LSG 
NinHSG NinLSG 

(HSG=high scoring group 
LSG=low scoring group 
N=number) 

Item discrimination is generally interpreted in the following way: 



D-Value 
>+.40 
+.20 to + .40 
- .20 to + .20 
<-.20 

Refer to Previous Example 



Direction 

positive 

positive 

non 

negative 



P = (8/10 -3/10) = 5/10 = .50 



Strength 

strong 

moderate 

moderate to strong 
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Interpretation: This item is positive and strongly discriminating. A larger number 
of high, rather than low-scoring students, correctly answered the item. For more 
information or further resources on this subject, please contact the Office of 
Institutional Research. 

Other Features to Consider 



The pattern of incorrect responses is another feature of item analysis. Ideally, each 
incorrect response should be selected by a higher percentage of low- than high- 
scoring students. In the previous example few students selected responses A and 
B; these options do not provide much information about student knowledge and 
should be revised. 

The relationships of difficulty and discrimination patterns to intended test purpose 
is a critical aspect of item analysis. The table below outlines how the majority of 
items should function for norm- and criterion-referenced tests. 



Table 3. Relationships Between Item Analysis Indices and Test Purposes 

Test Purpose 



Item difficulty 



Norm-referenced 
Depends on the number of 
response choices; generally 
around .50 



Criterion-referenced 
Depends on level of mastery 
require, generally < or = .75 



Item discrimination 



Should be > or = .20; should 
be answered correctly by 
students with high total scores 
and incorrectly by students 
with low total scores 



Should be non- 
discriminating between +.20 
and -.20 



Analyzing the Quality of a Test 

Analyzing the quality of a test involves an examination of validity and reliability 
characteristics of the test. Some basic definitions and critical concerns are 
reviewed below. 




52 

49 



Validity; 



• Is the most important quality of a test 

• Does not refer to the test itself. 

• Generally addresses the question: “Does the test measure what it is intended to 
measure?” 

• Refers to the appropriateness, meaningfulness, and usefulness of the specific 
inferences that can be made from test scores. 

• Is the extent to which test scores allow decision-makers to infer how well 
students have attained program objectives. 

Table 4. Two Important Types of Validity 



Type 


Definition 


Validitv Threatened 

by 


Prevention & Checks . 


Content 


Extend to which a test 


Poor relationship 


Use and review of test 




adequately samples 


between test items and 


blueprint. 




program objectives. 


program objectives. 




Construct 


Extent to which a test 


for obiective tests 


Use of good item 




measures the amount 
learned and not some 


poorly constructed 
items that measure test- 


writing practices. 




other extraneous 


taking skill rather than 


Development of good 


Reliability; 


variable. 


mastery of material. 

for other test formats: 
problems with raters 
and scoring procedures 


rating criteria: train for 
& check on raters' 
consistency. 

Check correlations with 
other related & 
unrelated information 
(i.e., ASSET arid 
GPA's). 



• is the degree to which test scores are free of errors of measurement due to 
things like student fatigue, item sampling, and student guessing? 

• of objective tests is threatened by poor item construction. 

• of other test formats is threatened by rater or scorer variability and inconsistent 
application of scoring/rating standards. 
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Chapter V 



Using Performance-Based Measures for Assessment 



• Types of Performance-Based Assessment 

Portfolios 

Performance Measures 

• Questions and Answers 
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Using Performance-Based Measures for Assessment 

}0iile; tests and surv^s remain the most popular assessment instruments, many 
departments are beginning to recognize the value of assessment that is based on 
student; classroom activities^ This chapter explores different types of performance^ 
based assessment activities and discusses ways to maximize their effectiveness. 

Perfonnance-rbased assessment is the process of using student activities, rather tiian 
tests or surveys, to assess skills and knowledge. Class assignments, auditions, 
recitals, projects, and others, while intended to evaluate the individual student, can 
be reviewed as a whole (using all or a sample), to evaluate the course. What does 
overall student performance on these assignments tell you about your course(s)? 
While performance-based projects can be designed specifically for assessment, you 
might consider using existing classroom projects as assessment tools. 

The last couple of years have seen remarkable growth in the use of portfolios and 
other types of performance-based assessments. These activities are being used in 
conjunction widi tests to provide a complete picture of student skills and abilities, 
rather than; simply relying on one test score. A student’s health, personal life, or 
even the weather can adversely affect test scores. Critics of testing also point that 
tests, partiQularly multiple choice tests, don’t provide sufficient opportunity for 
students to think through what they are doing, or to want to do their best. 

Portfolios and other forms of performance-based assessments, on the other hand, 
inyite the student to show his or her “best” work (Belanoff & Dickson, 1991, p. 
xvi). 

Types of Performance-Based Assessment 

• Portfolios 

•. Performance measures 

• Unobtrusive measures (see Chapter VII) 

Portfolios 

Collected examples of student work over time. 

Thou^ the use of student portfolios has a long history in disciplines such as art, it 
is rapidly gaining popularity in other areas, particularly in the assessment of 
writing skills. While portfolios are frequently used to evaluate an individual 
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student’s progress, they are also useful in allowing a department to take a critical 

look at overall performance of students in the program. 

Possible items to include in portfolios: 

• Exams (multiple choice and essay) 

• Research papers 

• Essays 

• Projects 

® Videotapes 

• Audiotapes 

Suggestions for Using Portfolios: 

• A department could keep portfolios on a sample of students in selected classes. 
Faculty members could then rate the portfolios according to the goals and 
objectives of the departmental outcomes. 

• Students could be asked to maintain portfolios as they progress through their 
general education courses. 

® To assess writing skills as a general education outcome, a department could 
choose to keep samples of student writing from all the courses in the 
department. Are students getting ample opportunity to write? What types of 
writing assignments are being required? How does the writing ability of 
students improve in a series of classes? Is there one particular class in the 
program that seems to have a significant impact on the writing skills of the 
students? 



Performance Measures 

Using examples of students ’ writing, presentations, or projects for assessment. 

Again, looking at overall student performance on these types of classroom 
assignmepts can tell you a great deal about your courses. You can learn a 
tremendous amount from just a few projects if you choose the right projects. 

Suggestions for using performance measures for assessment: 



* A GdmmSii writing assignment, such as ah essay or rese^eh papeh can be used 
fS examine more than one aSpeGt Of a course. Ah initial reading of a paper can 
assess students ’ uhderstaiiding of the course cOhterit. A secOrid reading could 
evaluate writing skills, while a third feadirig eOuld explore critical thinking 
skills. Similaflyj a computer project can be used to evaluate both students’ 
kriowledge of COiifse Content as well as the level Of eOmptiter cOhipetehcy. 

^ Samples of Students’ ait of photographic work could be displayed on a 
classroom walk Faculty members can then examine the work against a set of 
criteria arid make jhdgmertts about the strengths and weakhesses of the 
program. 

* If students are required tO perform, as in a drama, music Of speech cOufse, the 
peffOriflanees could be videotaped. A team of faculty members cOiild review a 
sample of the performances against a set of criteria. 

Qiiesfidhs & Answers oh Perforiiiance-BaSed Assessment 

Q) What is the fifSt step in establishing pefformance^based assessment? 

A) Begiri by asking faculty membefs to answef two questions: 

• What Outcomes do we value? 

* iiow wOuld we identify a successful student in ouf COurse(s)? 

Aubrey FOffestj EiifeCtof Of Assessment and Educational MeaSiifemeht at EmpOria 

State Uhivefsity (Fofrest, 1990^ p. 5), makes the following siiggestioris fOr 

pfiOritizihg dutcomes for portfolios, but his remarks have general applicability to 

Othef forms of assessment as well: 

1. Begin by collectirig a repfesentative fahge of student wOrk - from marginal to 
Outstahdihg -- ahd deterthirie what leafnihg appears to be taking place and what 

h)r a discussion Of what the most important 
ihtended outcomes ought to be. 

2. Identify a gfoiip Of “exemplary Students,” collect their course Work or 
ObSerVarices of theif perfoimahceSj ask the students to describe what 
expefiences in the cOiirSe helped them the most and then dO art analysis of the 
interviews and course wOrk. 
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3. Concentrate on one or more outcomes of your program for which portfolios or 
performances seem most useful or appropriate. Some outcomes, such as 
reading or computation skills, may have been adequately assessed already. 
Higher-order or more complex outcomes may be more appropriately assessed 
by a performance-based process. 

4. Base the evaluation on an existing official statement of intended outcomes such 
as your goals and objectives for the course. Build your assessment around this 
detailed statement of goals and guidelines. 

Q) Who needs to be involved in performance-based assessment? 

A) This will vary with departments. Many assessment techniques, by their very 
nature, don’t allow many opportunities for discussion and debate among faculty. 
Frequently these activities are designed and implemented by one, maybe two, 
faculty members. Performance-based assessment, though, as one writer puts it; 
“practically begs for conversation” (Belanoff & Dickson, 1991). Getting faculty 
members together to discuss what constitutes, for example, a successful student 
can be thought provoking and enormously rewarding. 

Q) What are the advantages of performance-based assessment? 

A) Assessment activities that are separate from the daily teaching routine of the 
department can be perceived by students and faculty as intrusions. Performance- 
based assessment builds on the daily work (assignments, exams, projects) of 
students and faculty. Probably the bluest problem faced by those conducting 
assessment activities is student response - not only getting students to participate 
in “voluntary” activities such as assessment tests and surveys, but getting them to 
do their best. While students are usually highly motivated to do their best when 
working on something that will count toward their grade, they are less likely to do 
so on an activity, which will not affect their grade. Using actual class projects 
means getting results from students who were probably motivated to do well. 

Q) Should we use students’ best work or choose from a range of grades? 

A) Actually either way is fine - the questions is, which will best suit the 
department’s purpose? Since students’ “best” will vary, you can learn a lot about 
your program from looking at those assignments which earned the highest grades. 
What differences do you see in these samples of work? Why are some students’ 
“best” works better than others? On the other hand, selecting from a wide range of 
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grades will say something as well. What is failing, average, or excellent work in 
ypnr opnrses? Is there commonality among each of these ranges that gives some 
cine as to who is doing What kind of work and why? 

iQ) Shonld we collect work from all our students or just a sample? 

A) This depends on the size of your student population as well as the type of 
project being used. If you have only one or two sections of thirty students each, 
then perhaps ynu could keep portfolios or do performance reviews for each of 
them, If, though, you have eight sections with anywhere from 40 to 90 students in 
each, a sampling of students is sufficient. 

Q) Who decides which items or projects go in portfolios or are used for 
performance assessment -- students or faculty? 

A) The type of assessment being done determines this. In some cases, faculty 
members will ask the students to select what they (the students) consider to be 
their beat work for inclusion in their portfolios. In other situations, the 
instructor may provide a list of class assignments or exams that are to be 
included in portfolios. The department may choose to create the portfolios 
themselyeSj ratiher than asking the students to do it. Either way, students 
probably have a right to know what you are doing and why. 

Q) Whe should see the materials in portfolios? 

A) This is, again, a matter for the department to decide, depending on what your 
purposes are. It could be a committee only who sees the materials, or it could 
be open to all faculty members, advisors, and even students. 

Q,) How do we know if our assessments are reliable? 

A) Aubrey Forrest suggests that reliability means the extent to which: 

1. The eyaluators agree on the analysis of the portfolios or performances. 

2, These same evaluators would make the same decisions at another time. 

3 . These evaluators, would agree with another group of evaluators. 
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Q) What are the standards for rating performance-based materials? 

A) Once the intended outcomes for the course have been identified, the faculty 
will need to set the standards for the evaluation of the portfolio materials or 
performance measures. For example, the faculty may decided that the course 
is doing what it is intended to do if 30% of the students’ materials or 
performance are rated at least 3 on a 6-point scale, 30% rate 4, and so on. A 
course might be judged successful if the average for the group is at least 3 on a 
6-point scale. The important thing is that there is consensus among faculty 
members. Performance-based assessment is most effective if faculty members 
agree on the intended outcomes of the course or project, as well as the 
standards set for the evaluation. 
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Chapter VI 



Uses of Institutional Data for Assessment 



• Using Student Data to Enhance Survey Data 

• Using Student Data to Enhance Test Data 



Uses of Institutional Data for Assessment 

This chapter explains some of the institutional and student data that are available 
for use within an assessment plan. Information that can be used to describe a 
particular class, to enhance survey data or to enhance test data is discussed. 

Various data are routinely collected at the college level that can enhance and 
further explain assessment data collected by departments, T\vo types of data exist: 
class and student. 

What departmental data is routinely available? 

Student credit hours - by course and by department 

Enrollments - by course and by department 

Faculty full-time equivalents - by course and by departnient 

Number of sections taught - by course 

Grade spread - by course and by quarter and by academic year 

Numiber of degrees produced - by department/program 

Class size ^ by course and by department 

Running Start enrollment - by administrative unit, course and by department 

Gender by Academic Year - by department 

Age by academic year - by department 

Age, gender, ethnicity by academic year - by department 

Gender by quarter - by department 

Ethnicity by year/quarter - by department 

Age by quarter - by department 

Age, gender, ethnicity - by quarter and by department 

Alumni survey data - by department 

Possible Questions for Using Data widi Course-Related Assessment 

• Is this a popular course? What is the enrollment trend for this course? 

• Are the students performing at the expected grade level? 

• Are the class sizes too large? 

• Are the classes being taught by full-time or part-time faculty? 

• Did the course affect program enrollment? 

• Does the course attract students to take other courses in the department? 
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Suggestions for Uses of Data with Program-Related Assessment 

® Is the program growing or shrinking? 

® What are the grades being given by faculty in the department? 

» What percent are retained in the (vocational) program? 

® What percent of students graduate from the (vocational) program? 

® What percent of students are employed from the (vocational) program? 

Available Student Data 

Basic Information 

Student name 

Gender 

Ethnicity 

Date of Birth 

Resident Status 

Citizenship 

Entry Data 

ASSET scores 

High School name 

Date of High School graduation 

Credits attempted 

Credits earned 

Credits transferred in 

Course enrollment data 

Grade point average 

Hours registered per quarter 

Degree attainment by program 

Year degree awarded 

Program 

Queirter entered program 
Purpose for attending 
Advisor name 
Major, if designated 
University of choice, if designated 






Using Student Data to Enhance Survey Data 

A department may decide to administer an attitudinal survey to students enrolled in 
Office Technology 201. To match survey responses with student database 
information, the survey will need to ask each student to provide their social 
security number keeping in mind that the human subjects policy requires letting the 
student know that giving his/her social security number is optional. The following 
tables are samples of possible correlations of survey responses with information 
obtained from the student database. 

Office Technology 201 
Survey Responses - Fall 1999 

Question: 

This course has shown me practical applications of using spreadsheets. 

Strongly Strongly Percent 



GPA Ranges 


Agree 


Agree 


Indifferent 


Disaeree 


Disaeree 


Total 


N 


NoGPA 


19.6 


62.7 


8.5 


9.2 




100 


153 


<1.0 


11.1 


68.9 


15.6 


2.2 


2.2 


100 


45 


1.0 -2.0 


4.5 


59.1 


31.8 




4.5 


100 


22 


2.0 -3.0 


6.3 


75.0 


18.8 






100 


16 


>3.0 


25.0 


50.0 


25.0 






100 


4 
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Question; 

This course has increased my understanding of spreadsheet theory. 



Credits Taken 
w/denartment 


Strongly 

Agree 


Agree 


Indifferent 


Disagree 


Strongly 

Disagree 


Percent 

Total 


N 


<10 


20.0 


30.0 


40.0 


10^0 




100 


10 


10-45 


26.1 


39.1 


21.7 


4.3 


8.7 


100 


23 


45-80 


5.7 


51.4 


25.7 


11.4 


5.7 


100 


70 


80+ 


5.3 


49.3 


25.3 


20. 




100 


75 



Using Student Data to Enhance Test Data 




A department may decide to correlate test results with information about the 
students who took the test. Again Social Security numbers may be needed to get 
the information. 

Math 101 

Final Exam - Fall 1999 




Mean Score 


Low Score 


High Score 


N 


All Students 


77.2 


40 


99 


565 


Gender 










Male 


75.3 


40 


99 


259 


Female 


78.9 


45 


99 


306 


Part Time 

1 


74.2 


40 


89 


225 


Full Time 


80.3 


61 


99 


340 


Designated Major 










Biology 


77.5 


60 


96 


47 


Business 


79.0 


40 


99 


106 


Education 


76.1 


40 


94 


75 


Arts 


75.0 


45 


90 


35 
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Chapter VII 

other Assessment Techniques 

• Interviews 

• Focus Groups 

• Unobtrusive Measures 

• Review of Curriculum Materials 

• Anecdotal Records^ Logs, and Journals 

• External Examiners 
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Other Assessment Techniques 

There are many assessment techniques that can provide usefiil information, 

particularly when used in conjunction with the more traditional methods described 

in the previous chapters. This chapter discusses several of these techniques and 

how they can be used to enhance your assessment efforts. 

Interviews 

Gathering information in a focused, one-on-one conversation. 

Use interviews when: 

• You want information on a broader scale than a paper and pencil survey might 
allow you to obtain. 

• You want to explore issues of concern to the department. 

• You want to get information to design a paper and pencil survey. 

• You want more personal contact with long-distance students than a mail survey 
would permit. 

Types of interviews 

• Standardized interview with closed responses: In this type of interview, a set of 
standardized questions is prepared and asked of each participant. The role of 
the interviewer is simply to ask the questions and record the responses. The 
only information that is gathered is Aat which is specifically asked for, almost a 
verbal version of a paper and pencil survey. . 

• Standardized interview with open responses: This type of interview also relies 
on a set of standardized questions, but the questions are designed to elicit open- 
ended responses and the participant is encouraged to talk at length on the items. 
The interviewer uses the questions to guide the interview. 

• Non-standardized interview: This last method is essentially a conversation 
between the interviewer and the participant in which they agree to discuss the 
participant’s views of the subject matter. There is no set of questions to be 
asked - the interviewer merely probes the participant on his/her opinions or 
perceptions on a particular issue or topic. 
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Suggestions for Using Interviews: 



• Faculty members can be interviewed on their perceptions of the strengths and 
weaknesses of the department or of a particular program within the department. 
The department could generate the questions for the interviews or suggest the 
topics that they would like to have covered. A person from outside the 
department could be called in to administer the interviews and summarize the 
results. 

• A telephone interview can be conducted with department/program alumni. 

• Graduating seniors can be asked to participate in an individual exit interview. 

Focus Groups 

A form of group interview is where a researcher supplies the topics and monitors 
the discussion. The emphasis is on group interaction to share insights and ideas. 

Use focus groups when: 

• You need to get feedback from a small group of people. 

• You need a non-threatening format for a discussion. 

• You want to facilitate discussion among a mixed audience, such as the students, 
alumni, and faculty of the department. 

Focus groups allow a small group of people to discuss at length, in a non- 
threatening environment, a pre-designated topic. The dialogue is usually led by a 
moderator who works to keep the discussion focused on the chosen topic. For 
assessment purposes, the subject of a focus group could be: the perceived 
effectiveness of a course or program; designing a department assessment plan; the 
implications of the results of assessment activities. 

Suggestions for Using Focus Groups: 

e Department faculty can keep notes on what and how students in their classes 
are learning and then share their ideas and experiences with their colleagues. 

• To facilitate faculty involvement in designing an assessment plan, faculty 
members can be asked to respond to such questions as: “How would we 
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identify a successful student in our course(s)?” or “What outcomes do we 
value?” 



o The department can ask a group of their graduating students to meet with 
faculty members to discuss the perceived strengths and weaknesses of the 
program. A^at did the students experience in the program? Where did they 
have problems? From what experiences did they gain the most? Iftheyhada 
chance to do it over, what would they do differently? 



Observation or keeping records of students ’ use of facilities and services. This 
form of assessment provides data that can be correlated with test scores and/or 

course grades. 



Use unobtrusive measures when: 
o You are interested in how students study. 

o You are concerned about the effectiveness of study sessions or other 
supplements to classroom activities. 

o You are focusing on the relationship between out-of-class behavior and in-class 
performance. 

While test scores and survey results can indicate how much students are learning 
and how they feel about that process, observing the behavior of students can tell 
you quite a bit about “how” they learn. What do students do outside the 
classroom? Which out-of-class behaviors and activities seem to lead to better in- 
class performance? 

Suggestion for Using Unobtrusive Measures: 

o An indicator of student participation in a course can be how frequently students 
are using services such as study sessions, computer labs, reserved supplemental 
readings in the library - even tracking class attendance can be useful. 

o A department can determine how frequently students attend museums, concerts, 
plays, recitals, lectures, etc. 
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• It may be helpful to a department to know how frequently students visit their 
program advisor, or other faculty members. Are successfiil students more 
likely to seek advice or guidance? Of interest, too, may be how often students 
in the department make use of facilities such as the Learning Opportunities 
Center or tutoring services. 

Review of Current Curriculum Materials 

Systematic review of course syllabi, textbooks, exams, and other materials. 

Use curriculum materials when: 

• You want to clarify learning objectives. 

• You want to explore differences and similarities between sections of a course. 

• You want to assess the effectiveness of instructional materials. 

Basic to any instruction are the materials used in the classroom - textbooks, 

handouts, exams, etc. A review of these provides invaluable information to 

enhance any assessment effort. 

Suggestions for Using a Review of Curriculum Materials 

• Faculty members can review the current course syllabi to determine if the 
course is meeting the goals and objectives that were established for it. 

• A review of the course exams for the different sections might reveal enough 
standardization of exam items to allow those exams to be used to assess the 
course. For example, five different instructors who are each designing an exam 
for their section of a course could be asking similar questions on common 
subjects. Those common items could be used as an assessment tool much the 
same as if a standardized exam had been administered to all five sections. 



. AmecdlotaE Records, Logs, amd Journals 

Maintaining records of classroom activities, students 'responses, or faculty 

impressions. 



Use anecdotal records when: 

o You are concerned with how students and faculty react to particular 
instructional methods. 

o You are piloting new textbooks or other materials. 

o You are interested in students’ perceptions of certain aspects of the course. 

While assessment usually relies on systematic collection of information, often the 

more informal observations prove useful. What is going on in the classroom? 

What works and what does not? 

Suggestions for Using Anecdotal Records, Logs, and Journals 

® Faculty members can maintain journals on their experiences in the classroom. 
If, for example, a new textbook is introduced, a journal can be kept to record 
the student’s responses to the book, as well as the instructor’s reaction. 

® Students can be asked to evaluate visiting speakers or instructors. 

® Faculty members can ask students to write reaction papers in response to 

classroom activities, videotapes, field trips, etc. Did students find the activities 
helpful? How could the activities be improved? 

o Areas that provide services, such as Student Services will find anecdotal 
records, logs and journals particularly helpful. 
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External Examiners 

Using an individual or group of individuals from outside the department/program 
to provide an objective assessment of a course or program. 

Use examiners when: 

• You need an objective observer of your program. 

• You want an assessment expert to provide insight and advice. 

Asking someone from outside the department or college to observe and respond to 

particular aspects of a program or course can yield valuable information that could 

not otherwise be obtained. 

Suggestions for Using External Examiners 

• A department may invite a member of the English faculty in as a consultant to 
provide insight into the department’s use of writing in the discipline. The 
consultant can answer questions such as: Does the program provide ample 
opportunities for students to write? How can the department increase those 
opportunities? How could the department effectively assess their students’ 
writing ability? 



• An external evaluator from another college can be invited to moderate a focus 
group or to assist a planning session on designing a departmental assessment 
plan. 
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Chapter VIII 

Reporting and Using Assessment Results 



• Uses for Assessment Results 

• Questions and Answers on Reporting and Using 
Assessment Results 
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Reporting and Using Assessment Results 

The purpose of this chapter is to discuss the various ways that assessment results 
can be reported and used by departments. 

When reporting the results of department assessment activities, the first questions a 
department needs to ask are: “>^o is our audience?” arid “What dp they warit tp 
know?” The kind of report expected for departmental outcomes assessment may 
be completely different from what a program professional aecreditatipn review 
may require. 

For some reports, it may be necessary to discuss the statistical analysis and die 
results. Other types of reports may simply require submitting the list pf ypur 
department’s outcomes, their measurements (how well you did), what ypur criteria 
for success was, and how you plan to use the results. 

Checklist - To whom are we reporting results? 

Accrediting bodies 

Division deans 

Federal agencies 

College curriculum committee 

Department planning committee 

Alumni 

Colleagues at other colleges 
Conference presentations 
Students 

Uses for Assessment Results 

A question frequently asked by department faculty members is “How can 
assessment data he used?” The following are suggestions for different ways that 
assessment information can be heneficial. 

Primary uses: 

• Accreditation reports and reviews 
General education assessment 
Curriculum review 
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• Assessment results can be distributed to faculty members to encourage a 
discussion of perceived strengths and weaknesses of the department or to 
generate ideas. 

• Requests to the Curriculum Committee 

Assessment data may accompany requests to a college curriculum 
committee for additional courses. For example, an alumni survey might 
reveal that the department’s graduates find themselves lacking a particular 
skill required for the market. The department could use the results of the 
survey to support a request for a new course to teach that skill and to give 
graduates a more competitive edge. 

• Requests for budget increases 

• Outcomes assessment report to Division Dean 

• Program review 

Secondary uses: 

• Recruiting 

Findings about the satisfaction of current courses or programs can be used to 
recruit new students to the program. 

• Program brochures. 

• Publications - assessment results can be shared with other institutions. 

• Career Services 

There is a growing interest in the successful marketing of graduates. 
Department alumni surveys can demonstrate to prospective employers why 
graduates from that program are more skilled and better qualified than 
graduates from similar programs at other colleges. This can be done on a 
departmental or institutional level, 
o Securing grants. 

Questions and Answers on Reporting and Using Assessment Results 

Q) How can we make sure that the results of our assessment activities are used? 

A) The first step in assuring that results are used is to share them with department 
faculty members. A written report could be prepared and distributed to the faculty. 
Faculty members who were involved in the assessment effort could present their 
findings at a department or division meeting. Departments could establish yearly 
planning retreats just for the purpose of discussing their assessment results. The 
results of your assessment activities can be used to generate a discussion among 



the faculty. What does the report tell you about your course or program? What 
possible changes are indicated? How could the department use the results to the 
best advantage? Depending on the department’s assessment needs, further reports 
or presentations could be prepared for different audiences. 

Q) What’s the best way to share our findings? 

A) The first step in sharing assessment results is usually to prepare a written report 
that can be distributed to both faculty members within the department and 
constituents external to the department. The type of assessment activity 
implemented and the audience for whom the report is prepared will 
determine the format of the report. For most audiences, a simple descriptive 
report is best. 

Q) What should be included in an assessment report? 

A) Depending on the audience, an assessment report should include enough 
information to answer four basic questions: 

1 . What did we do? 

2. Why did we do it? 

3. What did we find? 

4. How will we use it? 

Q) In reporting what we found, do we need to report actual data or the 
interpretation of the data? 

A) This depends on your audience. Some Deans may require the department to 
record actual findings from outcomes assessment projects. Other reports may 
call for a summary of findings - the implications of your assessment results. 

Q) Do we have to report everything? 

A) In most circumstances the department will choose what is to be reported. In 
addition to the expected results of an outcomes assessment project, there are 
often unexpected results. While you may be required to report the expected 
results (i.e., placement rate of graduates), the unexpected results may be useful 
only to the department. For example, an alumni survey may reveal that 
graduates are getting jobs in one particular type of firm or company. This 
information could allow the department/program to zero in on what job skills 
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their graduates have that are making them so marketable with these companies. 
While useful to the department, this information may not be of interest to 
constituents such as accreditation teams or college curriculum committees. 

Q) What is the best way to display assessment results in reports? 

A) For most purposes, if survey results need to be visually displayed, simple 
frequencies will suffice - tables that indicate the number and percents of 
responses for each category of the survey question. Test and survey results 
that are cross-classified with information obtained from the student database 
such as GPA or ASSET scores can also be presented in tables. For more 
detailed reports, you can use graphs or pie charts to illustrate findings. As with 
other aspects of assessment, the types of visual displays you use depends on 
the kind of assessment project and the audience for whom it is done. 

Q) How long do we need to keep our results? 

A) It is a good idea to keep assessment results for a few years. Departments often 
find it useful to do longitudinal studies of assessment projects. What trends or 
patterns are revealed when looking at responses from alumni or graduates over 
time? As the job market, the economy, technology, or even the department 
itself changes with time, so will the results of assessment activities. Also, 
many assessment evaluations are on cycles. The Outcomes Assessment reports 
are yearly. Accreditation is on a 10 year, five year, two year (CBC’s next 
report due), or in some cases one year. Similarly, some departments do alumni 
surveys annually, while others may prefer to do them every five years. The 
Office of Institutional Research does an annual alumni survey on the previous 
year’s graduates. To know what has been done before, to learn from past 
mistakes, or to do longitudinal studies, it may be advisable to keep the results 
of assessment projects for a period of three to five years. 
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B. Worksheet to Determine Linkages between 
Purpose Statement, Departmental Outcomes, 
Results, and Use of Results 

C. Examples of CBC Departmental Outcomes 
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D. Blank Departmental Outcomes Assessment 
Forms 

£. Assessment Timeline for Gathering and 

Presenting Assessment Reports 
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Appendix A 

Office of Institutional Research and Marketing Surveys 

This is a brief summary of college-wide surveys administered by the Office of 
Institutional Research and Marketing The results from these surveys may provide 
departments with information that can be used in conjunction with the 
department’s assessment activities. 

Alumni Survey for AA Graduates 

This survey is administered yearly to the previous year’s graduates and includes 
demographic, instructional, student services related, student satisfaction, transfer 
information, and general education information. If a department would like to 
include a separate list of questions for a sub-set of students, contact the IR Office. 

Alumni Survey for AAS Graduates 

This survey is administered yearly to the previous year’s graduates and includes 
demographic, instructional, student services related, student satisfaction, and 
employment information. If a program would like to include a separate list of 
questions for a sub-set of students, contact the IR Office. 

Employer Survey 

A survey is administered yearly to local employers and also to those employers 
noted by alumni survey respondents. This survey gives an overall picture of 
employer . satisfaction. If a program wishes to add questions to a sub-set of 
employers, contact the IR Office. 

Employment Information 

Annually the IR Office does an employer match using information from 
Employment Security to' determine which CBC students are employed, where they 
are employed and their wages. This information is provided to each vocational 
department lead for use in determining their departmental outcomes measures and 
how successful they were in meeting their criteria for success. 
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Community College Student Experience Questionnaire 

This survey was administered in the fall of 1996, the fall of 1999, and afterwards 
on a periodic basis. This survey measures how involved students are in their own 
learning by asking questions on instruction, use of the library, student activities, 
estimate: of gains in general education, and college specific questions. Infoimation 
on results of this questionnaire is available on the CBC Web page under Office of 
Institutional Research and Marketing. 
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Worksheet to Determine Linkages between Purpose 
Statement, Department Outcomes, 

Results and Use of Results 
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Example of Linkage between Expanded Statement of 
Institutional Purpose, Departmental/Program lutended 
Outcomes/Objectives, Results, and Use of Results 
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Appendix C 



Examples of CBC Departmental Outcomes Assessment 
Forms (Carpentry and English) 
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ASSESSMENT meO'R©;' 

FQRiGarpentry 

(Department/Program); 

1997-98 ^ 

(Period Covered) . 

Jepartmental/Progr^ Statement of Purpose 

■ Mission: The Columbia. Basin College Carpentry Program will provide skill, training for. entry level 
.work,, provide re-trainihg.or brush up for those seeking employment or those Gurrently. employed. 



ntended Outcomes ^ 

1. At; the. end’ of the first ye^, students, will have gained the knowledge md skills necess^ for entry 
level job placement in the. carpentry industry. 



; 2. Employers of the C^entry Program graduates in the Tri-Cities area will be satisfied with the 
training received by their employees. 



3 . At the end of the second year, students will have gamed the knowledge and skills necessary to 
enter into, advanced carpentry employment. 



4. 90% of the students, after leaving the Carpentry Program at CBC will be employed. 



Form B 






ASSESSMENT RECORD 
English and Literature Departments 
1998-99 Academic Year 



Intended Outcome 

NOTE: There should be one form C for each intended outcome listed on form B. 

English Composition 

1. Students will use and understand the writing process to write clearly and effectively; students will 
also 

• write in a variety of forms for different audiences and purposes 

• effectively evaluate the quality their own work and that of others. 



First ' 

a. Means of Assessment «& Criteria for Success: 

• 80 percent of students actively enrolled in college-level writing courses will complete the courses 

with a 2.0 GPA or higher' 



a. Assessment Results: 


N/A 




a. Use of Results: 


N/A 


Second 



b. Means of Assessment & Criteria for Success: 



® The Department will revise the syllabi for English 101 and English 201 yearly to reflect 
the intended learning outcomes. 



b. Assessment Results: 

N/A 



b. Use of Results: 

' • ^ 

Form C 



1 This benchmark standard is based on a historical trend of CBC students’ past performance and a state average of 
core-course completion rates, as measured and reported by the State Board for Community and Technical Colleges, 
Resolution 97-09-37. The State Board is drafting legislation to increase the rate to 85% of students who 
successfully complete English and math classes. 
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^ ASSESSMENT RECOW) 

English and Literature Departments 
1998-99 Academic Year 

Intended Outcome 

NOTE: There should be one form C for each intended outcome listed on form B. 

Literature 

1 . Students will read and understand college-level literary texts; students will also: 

• share knowledge gained from reading with others \'ia class or group interchange to 
improve understanding 

• use a variety of reading strategies to understand what is assigned. 



First ; ■ 

a. Means of Assessrnent & Criteria for Success: 

• 80 percent of students actively enrolled in literature courses will complete the courses with, a 2.0 

GPA or higher.^ 



a. Assessment Results: 

N/A 



a. Use of Results: 


N/A 


Second 





b. Means of Assessment & Criteria for Success: 

N/A 
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successfully complete English and math classes. _ 
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ASSESSMENT RECORD 
Enghsh and Literature Departments 
1998-99 Academic Year 

Intended Outcoihe 

NOTE: There should be one form C for each intended outcome listed on form B. 

English Composition 

2. Developmental writing students will demonstrate the wnting skills necessary to succeed in 
college-level writing courses. 



First ^ ^ ^ 

a. Means of Assessment & Criteria for Success: 

• Students enrolled in English 98 will take a common final essay exam, which will be evaluated 
holistically by the English faculty. 

• 70 percent of students actively enrolled in English 98 will successfully complete the course.^ 

• The Department will revise the syllabus for English 98 yearly to reflect the intended learning 
outcomes. 





a. Assessment Results: 


N/A 




a. Use of Results: 


N/A 


Second 



b. Means of Assessment & Criteria for Success: 



• 60 percent of students who pass English 98 will pass English 101 with a 2.0 GPA or higher. 

Students’ grade results will be obtained from the registrar and calculated after the first college- 
level writing class. 





b. Assessment Results: 


N/A 




b. Use of Results: 


N/A 



Form C 



2 This benchmark standard is based on a historical trend of CBC students’ past performance and a state average of 
core-course completion rates, as measured and reported by the State Board for Community and Technical Colleges, 
Resolution 97-09-37. The State Board is drafting legislation to increase the rate to 85% of students who 
successfully complete English and math classes. 
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ASSESSMENT RECORD 

EngHsh and Literature Departments 
1998-99 Academic Year 



Intended Outcome 

NOTE: There should be one form C for each intended outcome listed on form B. ' 

Literature 

2. Students will transfer reading comprehension and analytical skills to other academic, personal, or 
professional situations. 



First ■ ' ' ' . ' 

a. Means of Assessment & Criteria for Success: 

• 75 percent of students enrolled in literature courses who receive a survey will complete a survey to 
measure quantitative and qualitative responses. 

• 50 percent of the students who complete the survey will positively indicate that they Can transfer 
reading and analytical skills to other academic, personal, or professional situations. 



a. Assessment Results; 

N/A 



a. Use of Results: 

N/A 



Second , . 

b. Means of Assessment & Criteria for Success: 

• The Office of Institutional Research will assist with analyzing the students’ surveys, and the 
results will be shared with the English faculty. 



b. Assessment Results: 

N/A 



b. Use of Results: 

N/A 



Form C 
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ASSESSMENT RECORD 
English and Literature Departments 
1998t99 Academic Year 
Intended Outcome ■ . . 

N OTE: There should be one form C for each intended outcome listed on form B. ■ 

English Composition 

3. Students will successfully transfer writing skills to other academic, personal, or professional situations. 



"irst ^ ^ ^ 

a. Meansof Assessment & Criteria for Success: 

• 75 percent of students enrolled in English 105 and English 201 will complete a survey to measure 
quantitative and qualitative responses. The surveys will be completed by students in class near the end of 
the quarter. 

• 50 percent of the students who complete the survey will positively indicate that they can transfer writing 
to other academic, personal, or professional situations. 

• The GPAs and completion rates of one group of students enrolled in a core course, and who has passed a 
college-level writing course, will be compared with that of another group of students without any prior 
college-level English or literature courses. The core courses involved in the comparison will be 
Introduction to Art, Sociology 101, and Geology 101. Winter quarter would be the best period to compare 
the groups who had completed college-level writing courses in fall quarter. ASSET and other 
standardized tests scores will be conridered. 



a. Assessment Results: 

N/A 



a. Use of Results: 

. N/A - . 

Second ^ 

b. Means of Assessment & Criteria for Success: 

® 60 percent of students who pass English 201 or 105 and who have enrolled at WSU Tri-Cities (fall 

semester 1 998) will complete a survey to measure quantitative and qualitative responses. The Office of 
Institutional Research will assist with analyzing the students’ surveys. English faculty will prepare and 
administer the survey. Results may also be obtained from the CBC Alumni Survey, which will be 
administered by the Office of Institutional Research. 



b. Assessment Results: 

N/A 



b. Use of Results: 

• The results will be shared with the English faculty to help modify the curriculum as needed. 



Form C 
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^SSESSMENOF RE€OW^ 

English: and Literature 

1998r99 Academic Year 

Intended Outcome 

NOME: There shouldbe one. form G. for each.intended outcome listed on, fonu B. 

\ English Composition 

X 4\. Smd.ents wilI.use;the;Gollege’s Writing Center, including the online. Writing Center, to obtain 
necessary, assistance; 



First • ^ 

; a^ Means of Assessment & Criteria; for Sue 

! 

i 20;percent of students; enrolled in, all; English composition classes will register and \york with a 
< tutor; (either- individually or in a workshop), at least once per quarter. 







•• 





a., Assessment Results.: 


N/A 








a. Use of Results; 

i 


N/A 






Second 









b. Means of Assessment & Criteria for Success; 



• Accurate tutpring logs will be kept, indicating the names of the students., tutors, and^ instructors; 
the classes and assignments pr projects; as well as the areas of tutpring assis,t^.9e provided. 





b. Assessment Results: 


N/A. 



















b. Use of Results: 



• Tutpring results will be shared at least twice per quarter with all English faculty via campus Ct 
mail, enabling instructors tp monitor their students’ progress. 



Form C 
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Appendix D 



Blank Departmental Outcomes 
Assessment Forms 



ASSESSMENT RECORD 
FOR 



(Departmeht/Program) 



(Period Covered) 
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ASSESSMENT RECORD 
English and Literature Departments 

1998-99 Academic Year 



DepartmentaiyProgram Statement of Purpose 



• The mission of the English Composition department is to provide students with college-level 
skills in critical readingvand thinking, methods of research and documentation, analysis and 
interpretation of various forms of discourse, as well as writing across academic disciplines and 
creative and technical fields. 

• The mission of the Literature department is to broaden students’ knowledge of human experiences 
and to provide students with a college-level understanding of literary works, including the 
historical and cultural contexts, the means and circumstances of production, the forms and 
conventions, the functions served, as well as various methods of textual criticism. 

• The combined mission of the English and Literature departments is to prepare students for either 
transfer to four-year colleges and universities or for entry into vocational or professional settings. 



FORM B 






Intended Outcoraw 

English Composition 

1 . Students will use and underst^d the writing process to write clearly ^d effectively; students will 
also 

•. \yrite in a variety of forms for different audiences and purposes 

• effectively evaluate the quality their own work and that of others. 

Literature 

I . Students will read and understand college-level litermy texts; students will also 

• share knowledge gained from reading with others via class or group interchange to 
improve understanding 

•. use a variety of reading strategies to understand what is assigned. 



English Composition 

2. Pevelopniental writing students will demonstrate the writing skilfr necess^ to succeed in 
college-level writing courses. 

Literature 

2. Students win successfully transfer reading comprehension and analytical skills to other academfr, 
personal, or professional situations. 

English Composition 

3. Students will successfully tr^sfer writing skills to other acadeniic, pereonal, or professional 
situations. 

English Composition 

4. Students will use the college’s Writing Center, including the online Writing Center, to obtain 
necessary assistance. 



ASSESSMENT RECORD 
FOR 

Carpentry 

(Department/Program) 

1997-98 

(Period Covered) 

Intended Outcome 

NOTE: There should be one form C for each intended outcome listed on form B. 

At the end of the second year, students will have gained the knowledge and skills necessary to 
enter into advanc ed carpentry employment. ^ 

First 

a. Means of Assessment & Criteria for Success: 



a. Assessment Results: 



a. Use of Results: 



Second ^ ^ 

b. Means of Assessment & Criteria for Success: Student employability by using data matches 
provided by the state board for community and technical colleges and Employment Security, plus the 
Alumni Survey. 



b. Assessment Results: 



b. Use of Results: 




Form C 
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ASSESSMENT RECORD 
FOR 

Carpentry 

(Department/Program) 

1997-98 

(Period Covered) 

Intended Outcome 

NOTE: There should be one form C for each intended outcome listed on form B. 

90% of the carpentry students that have completed two years in the Carpentry Program will 

be employed in the carpentry trade. . ■ 

"irst . ' 

a. Means of Assessment & Criteria for Success: Six months after the carpentry students have 
completed two years in the carpentry program, student employment will be provided by busing data 
matches provided by the state board for conimunity and technical colleges and Employment Security, 
plus the Alumni Survey, 75% of students who exit the program after one year with a certification will 
be employed within one year. 



a. Assessment Results: 



a. Use of Results: 



Second . . 

b. Means of Assessment & Criteria for Success: 



b. Assessment Results: 



b. Use of Results: 




S7 



Form C 



ASSESSMENT RECORD 
FOR 

Carpentry 

(Department/Program) 

■ 1997-98 

(Period Covered) 



Intended Outcome 

NOTE: There should be one form C for each intended outcome listed on form B. ■ 

At the end of the first year, students will have gained the knowledge and skills necessary for entry 
level job placement in the carpentry industry. 



First 

a. Means of Assessment & Criteria for Success: Student will demonstrate a marketable skill level 
by performing carpentry tasks well enough to be employable at entry level positions. 



a. Assessment Results: 



a. Use of Results: 



Second 

b. Means of Assessment & Criteria for Success: Student employability by using data matches 
provided by the state board for community and technical colleges and Employment Security, plus the 
Alumni Survey, 75% of the students who exit the program after one year will be employed within one 
year. 



b. Assessment Results: 



b. Use of Results: 



Form C 
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ASSESSMENT RECORD 
FOR 

Carpentry 

(Department/Program) 

1997-98 

(Period Covered) 



Intended Outcome 

NOTE: There should be one form C for each intended outcome listed on form B. 

Employers of the Carpentry students will be satisfied with the training received by their 
employees. 



"irst 

a. Means of Assessment & Criteria for Success; 80 % of the employers who respond to an employer 
survey conducted every three years by the college vrill respond that they are satisfied with the training 
received by their eiriployees. 



a. Assessment Results: 



a. Use of Results; 



Second ' ■ 

b. Means of Assessment & Criteria for Success; The Carpentry Program Advisory Committee will 
be surveyed yearly to determine their satisfaction with the Carpentry Program curriculum. 



b. Assessment Results: 



b. Use of Results: 



Form C^ ^ 



Appendix F 



References and Further Reading 
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Appendix £ 



Timelines for Gathering and 
Reporting Assessment Activities 
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Appendix E 

TIMELINES FOR GATHERING AND REPORTING ASSESSMENT 

ACTIVITIES 



Fall Quarter 

Gather data on previous year’s outcomes as determined by the designated 
measurements. Determine if criteria for success met your standard. As a 
department, review results and decide if any action is needed to reach department 
goals. Review outcomes to decide if they are continuing to meet your 
department’s needs. If they are not, create new outcomes, measurements and 
criteria for their success. 



Winter Quarter 

Present outcomes report, including information on measurements and if criteria for 
success were met to your Division Dean. This can be accomplished either by a 
written report or by using Form C of the Departmental Outcomes Assessment 
forms. Use information from this report, if appropriate, in your departmental 
budgeting process. If additional resources are indicated from your assessment, 
this data should be used to support the increase in funding. 

Gather data if your plan calls for quarterly assessment. 



Spring Quarter 

If your assessment plan calls for gathering data quarterly, data would continue to 
be gathered on those items. 



1G3 

er|c 



82 





Pasco Campus 

2600 North 20'^ Ave. 

Pasco, Washington 99301-3379 
Phone: (509) 547-0511 
FAX: (509) 546-0401 
TDD/fTY: (509) 546-0400 

CBC Richland Campus 

1011 Northgate Drive 
Richland, Washington 99352-3540 
(509) 946-9669 

Chase Centre 

1600 N. 20'^ Ave, 

Pasco, Washington 99301 
(509) 547-0511, ext. 2222 

Access CBC on the Internet: 

www.cbc2.org/ 

Columbia Basin College is accredited by the 
Commission on Colleges of the Northwest 
Association of Schools and Colleges. 

Columbia Basin College is a 
federal A A/EOE institution. 
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